System and methods for cuing visual attention

ABSTRACT

A system and methods for cuing visual attention using one or more sensory cues. Sensory cues, such as visual and touch cues, are used to determine an area neglected by a user and presented on a screen display to cue the user to direct his/her attention to that neglected area. Sensory cues are independent of the program content—medical image, airplane pilot simulation, security x-ray pictures—conveyed on the screen display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 62/382,539 filed Sep. 1, 2016.

FEDERALLY-SPONSORED RESEARCH OR DEVELOPMENT

This invention is made with government support under N00014-09-1-0069awarded by the Office of Naval Research. The government has certainrights in the invention.

FIELD OF THE INVENTION

The invention relates generally to a system and methods for cuing visualattention. More specifically, the invention relates to a system andmethods that uses attention input data, for example characteristics ofan individual's gaze on a display or location of a cursor correlated toa mouse input device, to cue visual attention to an area of a screendisplay that is neglected by the individual. Advantageously, anindividual may more easily manage visual attention to information on oneor more areas.

BACKGROUND

The vast majority of people are poor multitaskers. To make mattersworse, some of those who score worst on measures of multitaskingperformance tend to perceive that they are better at multitasking, witha negative correlation between perception and ability in large studies.These issues are particularly important, since in every day work-life,multitasking may often be necessary or efficient for a variety of humanlabor. Multitasking is directed to the cognitive process ofconcentrating on a specific sensory stimulus (i.e., attention to thestimulus).

A job may require an individual to direct attention to multiple tasksgiven the individual's responsibilities. For example, an individual maybe required to view displays on several monitors and focus on certaininformation. However, the individual may neglect viewing one or moredisplays or areas on some displays and miss some information.

Interactions with partially autonomous processes are becoming anintegral part of human industrial and civil function. Givensemi-autonomy, many such tasks can often be monitored by a user at onetime. In the course of operating a computer or vehicle, a single humanmight manage multiple processes, e.g., search and rescue type mobilerobots, performing medical supply distribution, patient checkup, generalcleanup, firefighting tasks, as well as process control with many dialsor readings, security or surveillance monitoring, or other forms ofhuman-based monitoring or tracking tasks. Generally, each automatedagent or process only needs intermittent supervision and guidance from ahuman to optimize performance, and thus a single user can remotelyoperate or supervise multiple entities, for efficiency of labor. Whencontrolling multiple automated processes at once, the user must decidehow to distribute attention across each task. Even if the operatorconducts the same type of task with each automated process, this form ofhuman-system interaction requires a multitasking effort.

Unfortunately, most people are notoriously poor multitaskers and canremain unaware of visually subtle cues that indicate the need for userinput. Further complicating the situation, individuals who perform worstat multitasking actually perceive they are better at multitasking,demonstrated by negative correlations between ability and perception ofability in large studies. To make matters worse, humans often naturallydevelop a plethora of biases of attention and perception. To addressmany of these issues, divided attention performance has been studied formany years. A further difficulty in multitasking is that brains relyheavily upon prediction and, fundamentally, are incapable of knowingwhat important information they have missed.

Eye tracking to ascertain point of gaze is a highly effective method ofdetermining where people orient their attention, as well as what theydeem important. Traditionally, eye tracking informed post-experimentanalysis, rather than helping users in the field in real-time. Forexample, a study might analyze optimal gaze strategies inhigh-performing groups, and then at a later date, train new users onthose previously discovered optimal search strategies. For example,studies have trained novice drivers' gaze to mimic experienced driverswith lower crash risk.

Alternatively, eye movement strategies can be employed to optimizereal-time task performance, since many eye-movements are capable ofbeing intentionally controlled. For those eye movements that cannoteasily be intentionally controlled, salient “pop-out” cues (e.g.,flashing red box around target) can reliably direct attention in a moreautomatic, bottom-up manner. As we discuss further, many eye trackingsystems have been developed for real-time control, with very fewattempting pure assistance, though none were both successful anddomain-general. Hence there is a need for such an assistive system.

Tracking a participant's eye movements while multitasking is anespecially good way to glean optimal cognitive strategies. Much work hasshown that eye tracking to determine point of gaze can reliably conveythe location at which humans' visual attention is currently directed.Locus of attention is a factor that can illustrate which of multipletasks a participant is currently attending to, as well as many otherdetails. Further, measuring where humans look tends to be highlyinformative of what is interesting to them in a particular scene, andcan be helpful for inferring cognitive strategies. Generally, gazeappears deeply intertwined with cognitive processes.

Multitasking principles also apply when managing multiple items inworking memory. For working memory, another cognitive construct that isdifficult to measure and discussed at length below, eye movementparadigms have revealed how visual search tasks can be interfered withwhen working memory is being taxed.

Though many paradigms have been developed to study multitasking usingeye tracking, most traditional applications of eye tracking are not usedin real time, but instead to augment training, or simply to observeoptimal strategies. For an example of training, post-experiment analysisof gaze data can be used to determine an attention strategy of thebest-performing participants or groups. Then, these higher-performingstrategies can be taught during training sessions at a later date.Implemented examples include educating health care professionals onvisual scanning patterns associated with reduced incidence of medicaldocumentation errors, and training novice drivers' gaze behaviors tomimic more experienced drivers with lower crash risk. As eye trackingmethods become more popular, they have been applied in the field ofhuman-computer interaction and usability, as well as human-robotinteraction, though in this area, guiding principles for optimal gazestrategies are still nascent.

Real-time reminders for tasks can improve user performance. Generally,real-time cuing of goals can speed or increase the accuracy ofdetection. Highlighting display elements in a multi-display may assistin directing attention, though eye tracking may often be critical toreliably automate such reminders for many tasks. As described above,there is little previous work developing real-time eye trackingassistance, with most research focused on training, evaluation, or basichypothesis testing. The real-time systems developed previously arelacking in domain-generality, utility, and flexibility. There is a needfor an assistive system and methods for managing multiple visual tasks,which is domain-general, transparent, intuitive, non-interfering,non-command, improves control (without replacing direct control), andadaptively extrapolates to a variety of circumstances.

Visual attention of the individual may be inferred from measuring thelocation of an individual's gaze on a display, for example, a graphicaluser interface on a monitor. Various technologies exist for measuringthe location of an individual's gaze and attention. A mouse cursorlocation may be used as an implicit measurement of attention. Forexample, a software program may be operated to identify the location ofa digital cursor on a graphical user interface positioned by a mouse andby implication an individual's gaze. Also, an optical sensor may measurethe location or duration of an individual's gaze. For example, asoftware program may calculate a vector between a pupil center and acorneal reflection to determine the location of an individual's gaze.

Eye-tracking systems measure the location of an individual's gaze on adisplay to determine whether the individual's visual attention isdirected on a certain area. Some conventional eye-tracking systemsdetermine whether the individual's visual attention is directed tospecific content in a certain area on the display. Hence these systemsare dependent on the content displayed on the screen. Other eye-trackingsystems only provide a visual cue, such as show a “warning”, to directan individual's visual attention. There is a need for a system andmethods to provide a cue to direct visual attention to an areaindependent of the content conveyed on the screen display.

The invention solves the above recognized needs by providing a systemand methods for providing sensory cue to direct visual attention to anarea neglected by an individual's gaze.

SUMMARY OF THE INVENTION

The invention is a system and methods for cuing visual attention.Advantageously, the invention may facilitate better management of anindividual's visual attention to one or more areas of a screen displayindependent of the program content conveyed or queued to be conveyed onthe screen display: the invention may be used with medical imagingsoftware, airplane pilot simulator software, security x-ray software,etc.

According to the invention, the system may include one or more inputdevices, sensors, computer processors, and output devices. In certainpreferred embodiments, the system permits a user to select a displayincluding one or more areas with at least one of the input devices. Theone or more sensors, such as a camera, eye-tracking device, touchsensor, and/or mouse, collect data from measurements of an individual'sgaze or touch (e.g., finger or mouse). At least one of the computerprocessors analyzes the attention input data to determine if anindividual's gaze/touch has neglected one or more areas for one or moreperiods of time and at least one of the output devices provides asensory cue to direct visual attention to the one or more areas anindividual's gaze has neglected for a period of time.

In preferred embodiments, the system is configured for cuing visualattention. A user initiates a software program stored on non-transitorycomputer readable medium so that the user may select one or moredisplays, such as a graphical user interface on a monitor, with one ormore areas each defined by a size, shape, and location on the one ormore displays. The user chooses one or more periods of time within whichan individual's measured attention must be on the one or more areas. Theuser chooses one or more sensory cues to direct visual attention to theone or more areas which have been neglected by the individual'sattention.

In certain embodiments, the system may be configured so that multiplesensors will measure an individual's attention. In certain preferredembodiments, the system may include sensors of the same type, forexample, two or more optical sensors. In certain preferred embodiments,the system may include sensors of different types, for example, a mouseand an optical sensor.

In certain embodiments, the system may be configured so that the outputdevice provides a sensory cue. The sensory cue may be visual and/oraudible. In certain preferred embodiments, a visible sensory cue mayinclude a change in color, hue, pattern, and/or shape. In someembodiments, a visible sensory cue may include a border around an area.In some embodiments, an audible sensory cue may include a certain sound,for example, sounds directing an individual to gaze at a specificdisplay.

In certain preferred embodiments, a display may be a graphical userinterface on a monitor. In some embodiments, the display may be adashboard. In some embodiments, the display may be a windshield. In someembodiments, the display may include at least one area configured tohave a different importance than at least one other area. For example,an area with a higher importance may have a shorter period of timewithin which an individual must gaze at the area before a sensory cue isprovided. In some embodiments, the importance of an area may changeafter the occurrence of a trigger event which may be configured by auser. For example, a user may configure one area's importance toincrease after an individual looks at a different area.

Attention of a user to one or more areas of a display screen is measuredto determine whether one or more areas have been neglected. Attentionmay be measured by the individual's gaze or controlled movement of aninput device. A sensory cue may be dependent on whether a specificindividual's gaze has not been at an area within a period of time.Furthermore, a sensory cue may be dependent on whether multipleindividuals' gazes have not been at an area within a period of time.

According to the invention, both a programming phase and an executionphase are provided for using one or more sensory cue to direct visualattention on a screen display. During a programming phase, an area on ascreen display where attention will be measured is defined. Sensory cuesare programmed to estimate attention. Parameters define sensory cuessuch as size, shape, color, overlay, etc. A user interface is accessedto define settings for each sensory cue. The user interface is used toenter a minimum time threshold. The minimum time threshold is theminimum time a user must pay attention to an area of a screen. The timea user pays attention to a particular area of a screen is determined byuse of an input device. The input device is used for acquiring attentioninput data, for example, a cursor within an area as directed by themouse input device controlled by a user or eye gaze within the region orframe as recorded by an eye tracker device. Each sensory cue defines anarea of the display screen and serves as an activation cue to direct theindividual's attention.

During execution, if a user does not pay attention to an area that meetsthe minimum time threshold, data is not accepted. If the attention inputdata recorded real-time meets the minimum time threshold, then the datais accepted and used to update in real-time a number of acceptedattention input data. The sensory cue not only defines an area of thedisplay screen for recording and accepting attention input data todetermine a neglected area of a screen display, but is also used topresent on the screen display to cue the user to direct their attentionto that neglected area. In order to determine if an area of the screendisplay is neglected, all numbers that total the accepted input data ofeach sensory cue are compared real-time. The sensory cue with thelongest duration of time attention input data was recorded within theframe as compared to other frames is assigned a neglected status. Themost neglected frame is that with the largest time since last attendedto, and the second most neglected frame is that with the second longesttime internal since it was last attended to based on the accepted data.All other frames are considered normal or regular. In other words, theframe with the longest time since last accepted data of attention incomparison to all frames is the most neglected frame. The associatedarea—an activation cue—is displayed on the screen according to theparameters selected during a programming phase.

The invention and its attributes and advantages may be furtherunderstood and appreciated with reference to the detailed descriptionbelow of one contemplated embodiment, taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated by way of example and not limitation in thefigures in the accompanying drawings, in which like references indicatesimilar elements and in which:

FIG. 1 illustrates a flow chart of method steps performed during aprogramming phase of the application.

FIG. 2 illustrates a user interface of an application that may be usedduring the sensory cue program phase.

FIG. 3 illustrates a flow chart of method steps for defining settingsfor frames during the programming phase of the application.

FIG. 4 illustrates a user interface that may be implemented to definesettings for each frame.

FIG. 5 illustrates a user interface that may be implemented to selectcolor characteristics of each frame type.

FIG. 6 illustrates a flow chart of method steps for adding frame recordsduring the programming phase.

FIG. 7 illustrates a user interface of an application that may beimplemented to add frame records.

FIG. 8 is a flow chart including method steps for executing the frameapplication.

FIG. 9 is a flow chart of the method steps performed during an executionphase for determining real-time neglected areas of the screen display.

FIG. 10A illustrates a screen display during the execution phase of theframe application.

FIG. 10B illustrates a screen display during the execution phase of theframe application.

FIG. 11 illustrates a screen display during the execution phase of theframe application.

FIG. 12 illustrates a computer system that may be used to implement themethods according to the invention.

FIG. 13 illustrates a cloud computing system that may be used toimplement the methods according to the invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The invention is directed to a system and methods for providing one ormore sensory cues to direct visual attention. A user's gaze on a screendisplay is monitored. Once it is determined that the user's gaze isneglecting an area of the screen display, a sensory cue prompts the userto direct their visual attention to that area of the screen display. Forpurposes of this invention, the sensory cue is a visual cue but anysensory cue is contemplated such as an audible cue or tactile cue.

The invention is directed to an application for selecting or creatingsensory cues, using the sensory cues to record attention input data todetermine a neglected area of a screen display, and presenting thesensory cue to cue the user to direct their attention to that area ofthe screen display.

According to the invention, there are two phases of the application thatprovide one or more sensory cues to direct visual attention: a sensorycue programming phase and a sensory cue execution phase. Sensory cuesare selected or created during the sensory cue programming phase. Duringthe execution phase, the sensory cues are used to determine areasneglected from a user's attention and are further presented on thescreen display as an activation cue to direct the user's attention tothat area.

The drawings and description are directed to a particular embodiment ofthe invention directed to a sensory cue in the form of a visual cue,otherwise referred to as an activation cue. More specifically, thevisual cue is in the form of a frame component defining an area orregion. The frame component comprises a window element and a borderelement, but any form of visual cue is contemplated. For example, thevisual cue could be any shape, size, or transparency. During anexecution phase, the frames are used to determine if a user isneglecting one or more areas of the screen display. The frame componentis illustrated according to programmed parameters on the screen displayto direct visual attention. Advantageously, the invention operates withno dependency on the program content conveyed or queued to be conveyedon the screen display: the invention may be used with medical imagingsoftware, airplane pilot simulator software, security x-ray software,etc. Hence the method is independent of the underlying application anddata, and can be used to reach optimal performance.

The programming phase of the invention is described and detailed inFIGS. 1-7. The execution phase is described and detailed in FIGS. 8-11.FIGS. 12-13 are exemplary systems that may be used to implement thephases of the invention.

FIG. 1 illustrates a flow chart 100 of method steps performed during aprogramming phase of the application. The method of FIG. 1 may beimplemented using the user interface 500 shown in FIG. 2.

As shown in FIG. 1, the programming phase of the application begins atstep 101. At step 103, either an existing file of frames is selected ora new file of frames is created. The input device selected is shown atstep 105. The input device is any device that can be used to provideattention input data, for example, a computer mouse, a touch screen, atouch pad, a stylus, a keyboard. Settings for the frames may then beselected at step 107. If settings are not selected at step 107, a newfile of frames may be created at step 150 (see FIG. 6).

FIG. 2 illustrates a user interface 500 of an application that may beimplemented to program the application for recording attention inputdata. As shown, an existing file may be opened by highlighting theexisting file in the documents window 501 and selecting the button 503.If a new file is created (see FIG. 6), button 505 is selected. To selectthe available input device—mouse or eye tracker—, its corresponding iconbubble 507 is selected. Settings button 509 is chosen to select settingsfor the frames.

FIG. 3 illustrates a flow chart 110 for directed to method steps fordefining settings for the frames during the programming phase. At step111, the minimum time threshold required to accept attention input datais entered. Attention input data is that which meets or exceeds theminimum threshold time, for example, a cursor within a frame as directedby the mouse input device controlled by a user or eye gaze within theframe as recorded by an eye tracker device.

Input data may be provided to the system using, for example, a computermouse, a touch screen, a touch pad, a stylus, a keyboard. Frameparameters are then determined. At step 113, a window fill parameter foreach frame is selected. At step 115, a border parameter is selected foreach frame. If desired, API server information can be entered at step117.

FIG. 4 illustrates a user interface 520 for implementing the method ofFIG. 3. Dependent upon the activity and related tasks, a time value isentered designating the minimum threshold time required to acceptattention input data within the frame. For example, analysis of a MRIscan of the heart may require viewing for 7 seconds to register a user'sgaze as attention input data. The minimum threshold time may be enteredin measurements of second, minutes, hours, etc. and may be entered usinga field 521 populated either using entry of a numerical value orscrolling +/− arrow controls 521A.

Window fill parameters 523 are defined using a field for each frame.Window fill parameters 523 include color and overlay in terms ofopacity, translucency, and transparency. Different colors/overlay may beused to designate each type of frame: a first neglected frame, a secondneglected frame and one or more regular (i.e., not neglected) frames.The neglected frames are determined based on the longest time sinceattention input data was recorded within the frame as compared to otherframes (see FIG. 9). The most neglected frame is that with the largesttime since last attended to, and the second most neglected frame is thatwith the second longest time internal since it was last attended tobased on the accepted data. All other frames are considered normal orregular. In other words, the frame with the longest time since lastaccepted data of attention in comparison to all frames is the mostneglected frame.

As shown, the most neglected frame is represented by field 523A directedto a dark opaque color. A second most neglected frame is represented byfield 523B directed to a medium translucent color. Each regular frame isrepresented by field 523C directed to a light transparent color. It iscontemplated that the colors may be reversed, for example, the firstneglected frame represented by a light transparent color.

Selecting any color field 523A, 523B, 523C may provide a pop-up screendisplay 530 as shown in FIG. 5. As shown in FIG. 5, the color for eachtype of frame—most neglected, second most neglected, regular—may beselected according to color tabs 531A, 531B, 531C, 531C, 531D, 531E. Tab531A includes a swatch grid 533 of various colors. One box of the gridmay be selected to designate the color of the frame type. The color foreach frame type may also be designated using tab 531B directed tobrightness in terms of hue, saturation, and value (HSV), tab 531Cdirected to lightness in terms of hue, saturation, and luminance (HSL),tab 531D directed to the additive color model red, green, blue (RGB), ortab 531E directed to the subtractive color model cyan, magenta, yellow,and black (CMYK).

Turning back to FIG. 4, border parameters 525 are chosen for the frametypes. Border parameters may include, for example, border thicknessand/or border style (solid, dashed, dotted). If desired, the frame maybe illustrated by only a border through the selection of a toggle box525A. Otherwise, the frame is illustrated by a window fill color,although it may be illustrated by both the border and window fill. Theborder thickness may be entered in measurements of millimeters,centimeters or any known measurement and may be entered using a fieldpopulated either using entry of a numerical value or scrolling +/− arrowcontrols 525B.

Application-programming interface (API) parameters 527 are selected todefine a set of programming instructions and standards for accessing aweb-based software application or tool. API server information can beinserted to point to a particular Internet Protocol (IP) address andport number to identify and locate the server from where the applicationis accessed. The IP address may be entered in a numerical format infield 527A and the port number entered either using entry of a numericalvalue or scrolling +/− arrow controls 527B.

In particular, the API server provides full control of the applicationincluding, for example, all options for settings for the frames, allsaved sets of frames and all alerting frames. Although the API serverprovides most information to the application for recording attentioninput data to determine a neglected area of a screen display, it iscontemplated that external applications and/or devices may be accessedto configure, start, and provide additional information to theapplication. It is also contemplated that external applications maycommunicate with the application asynchronously, with each handling adifferent portion of the application.

FIG. 6 illustrates a flow chart 150 of method steps for adding framerecords to an application during the programming phase. The method ofFIG. 6 may be implemented using the user interface 550 shown in FIG. 7.

As shown by step 151 in FIG. 6, a frame name is entered to identify therecord. The initial location of the frame is entered according to x, ycoordinates of the screen display at step 153. At step 155, the initialsize of the frame is defined in terms of width and height dimensions. Atstep 157, the frame is added as a file to the server database. After theframe is added, the frame is assigned a color for easy identification atstep 159. At step 161, the format—overlay, value, weight—of the framecolor is adjusted. At step 163, the frame record is saved.

FIG. 7 illustrates a user interface 550 for adding a frame record to anapplication for recording attention input data to determine a neglectedarea of a screen display. Field 551 is provided for identifying therecord by name including, for example, an alphanumeric combination. Theinitial location of the frame is entered according to x, y coordinates553. Values for the width dimension and height dimension for the initialsize of the frame is defined in fields 555. Once the frame record isadded as a file to the server database, such as by selecting button 557,the frame is automatically assigned a color for easy identification. Thecolor identifying the frame appears in a pie chart diagram 559 andcorresponding values for overlay 561A, weight value 561B, weightpercentage 561C are presented in configuration table 561. Selecting thedelete icon 561D removes the record from the database. In addition, theframe may appear on the screen display and can be manipulated by thecursor to be re-sized and moved. Accordingly, if the frame is re-sizedor moved on the screen display, the x, y coordinates 553 of the locationas well as the width and height dimensions 555 of the display 550 areeach automatically updated. It is also contemplated that the pie chartdiagram 559 may be manipulated to edit the related values of theconfiguration table 561. For example, a value may be entered in theweight value field 561B or arrow 560 can be manipulated to drag theedges of each pie slice to automatically adjust the weight percentage561C of the frame as listed in the configuration table 561. Adjustingthe weight percent increases or decreases the weight of the framewithout adjusting the weights of all other frames. Hence, the totalweight of all the frames may increase or decrease, instead of remainingthe same. Button 563 is selected to save the set of frames as a file tothe server database. Before executing the application, an alertindicator 564A and/or record indicator 564B may be selected. If thealert indicator 564A is selected, the frames are visible to the userduring execution of the application. If the alert indicator 564A isdeselected, the frames are not shown to the user. If the recordindicator 564B is selected, a detailed recording of the frames' usage isgathered. If the record indicator 564B is deselected, the frames' usageduring execution of the application is not recorded. The frameapplication is executed upon command, i.e., selection of graphic icon565.

FIG. 8 is a flow chart 180 including method steps for executing theframe application. At step 181, a user selects the one or more framesfrom a set of frames stored on a database for participation duringexecution of the frame application. Upon selection of the frames forparticipation, the frame application is executed.

FIG. 9 illustrates a flow chart 200 of the method steps performed duringthe execution phase for determining real-time neglected areas of thescreen display. At step 201, each frame is set to a zero value. At step203, attention input data is recorded. Data is accepted automatically atstep 204 when it meets the minimum threshold time (see FIG. 3). At step205, a number value of accepted data for each frame is continuouslyevaluated in real-time. All number values of each frame are compared atstep 207. Based on the comparison, neglected frames are selected at step209. The most neglected frame selected is that with the longest durationof time measured from the previous acceptance of input data incomparison to the other frames and the second most neglected frameselected is that with the second longest duration of time measured fromthe previous acceptance of data. All other frames are considered normalor regular (i.e., not neglected). In other words, the frame with thelongest duration since accepted data in comparison to all frames is themost neglected frame. At step 211, a visual sensory cue is activated forthe neglected frames. At step 213, one or more visual cues are displayedaccording to each frame's defined parameters (see FIG. 3) drawingattention to the neglected areas.

FIG. 10A, FIG. 10B, and FIG. 11 are directed to screen displaysillustrating visual sensory cues or activation cues during execution ofthe frame application. According to FIG. 10A and FIG. 10B, an eyetracker device is used to determine a number of times attention inputdata is accepted within each frame, i.e., the eye tracker accepts auser's gaze that meets the minimum time threshold. The neglected framesare determined real-time as discussed in FIG. 9.

The screen display 300 in FIG. 10A illustrates all three frame types:first or most neglected frame 301, second neglected frame 303 andregular or “not neglected” frames 305. Only the border element of frames301, 303, 305 is shown in FIG. 10A. The border element is illustrated asa solid, thin border as defined by the border parameters. FIG. 10Aillustrates the frame as illustrated through the selection of toggle box525A as described in FIG. 4. Without selection of of the toggle box525A, the window element illustrates the frames as shown in the screendisplay 320 of FIG. 10B, although it is contemplated that the frames maybe illustrated by both the border element and window element. FIG. 10Billustrates visual sensory cues during execution of the frameapplication in which the frames are presented according to the windowparameters selected. As shown, neglected frame 321 is shown as a darkopaque color, second neglected frame 323 is shown as a mediumtranslucent color and the normal frames 325 are shown a lighttransparent color.

FIG. 11 illustrates a screen display illustrating visual sensory cues oractivation cues during execution of the frame application. According toFIG. 11, a cursor's position is used to determine a number of timesattention input data is accepted within each frame, i.e., the cursor islocated within a frame for a duration of time that meets the minimumtime threshold. The cursor is positioned within a frame as directed byan input device controlled by a user. The neglected frames aredetermined real-time as discussed in FIG. 9. The attention input data isaccepted within each frame provided the cursor remains within a frame tomeet the minimum time threshold. FIG. 11 illustrates that frames may behidden upon a user directing visual attention to that frame. As shown,screen display 340A includes neglected frames 341A, 343A and regularframes 345A. A cursor 350A shown at screen display 340A is moved so thatthe cursor 350B is positioned over the neglected frame 343B in screendisplay 340B. As a result, the frame 343B is hidden from the user whenthe user indicates that he or she is providing attention input data tothat frame.

The system and methods of the invention were enabled in experimentsfurther detailed and described in the papers: (1) Human Strategies forMultitasking, Search, and Control Improved via Real-time Memory Aid forGaze Location, P. Taylor et. al, Front. ICT, 7 Sep. 2015 and (2)Eyeframe: Real-time Memory Aid Improves Human Multitasking ViaDomain-General Eye Tracking Procedures, P. Taylor et al., Front. ICT, 2Sep. 2015, both of which are incorporated by reference.

FIG. 12 illustrates a diagram of a system of which may be an embodimentof the invention. Computer system 600 includes an input/output interface602 connected to communication infrastructure 604—such as a bus—, whichforwards data such as graphics, text, and information, from thecommunication infrastructure 604 or from a frame buffer (not shown) toother components of the computer system 600. The input/output interface602 may be, for example, a display device, a keyboard, touch screen,joystick, trackball, mouse, monitor, speaker, printer, Google Glass®unit, web camera, any other computer peripheral device, or anycombination thereof, capable of entering and/or viewing data.

Computer system 600 includes one or more processors 606, which may be aspecial purpose or a general-purpose digital signal processor configuredto process certain information. Computer system 600 also includes a mainmemory 608, for example random access memory (RAM), read-only memory(ROM), mass storage device, or any combination thereof. Computer system600 may also include a secondary memory 610 such as a hard disk unit612, a removable storage unit 614, or any combination thereof. Computersystem 600 may also include a communication interface 616, for example,a modem, a network interface (such as an Ethernet card or Ethernetcable), a communication port, a PCMCIA slot and card, wired or wirelesssystems (such as Wi-Fi, Bluetooth, Infrared), local area networks, widearea networks, intranets, etc.

It is contemplated that the main memory 608, secondary memory 610,communication interface 616, or a combination thereof, function as acomputer usable storage medium, otherwise referred to as a computerreadable storage medium, to store and/or access computer softwareincluding computer instructions. For example, computer programs or otherinstructions may be loaded into the computer system 600 such as througha removable storage device, for example, a floppy disk, ZIP disks,magnetic tape, portable flash drive, optical disk such as a CD or DVD orBlu-ray, Micro-Electro-Mechanical Systems (MEMS), nanotechnologicalapparatus. Specifically, computer software including computerinstructions may be transferred from the removable storage unit 614 orhard disc unit 612 to the secondary memory 610 or through thecommunication infrastructure 604 to the main memory 608 of the computersystem 600.

Communication interface 616 allows software, instructions, and data tobe transferred between the computer system 600 and external devices orexternal networks. Software, instructions, and/or data transferred bythe communication interface 616 are typically in the form of signalsthat may be electronic, electromagnetic, optical, or other signalscapable of being sent and received by the communication interface 616.Signals may be sent and received using wire or cable, fiber optics, aphone line, a cellular phone link, a Radio Frequency (RF) link, wirelesslink, or other communication channels.

Computer programs, when executed, enable the computer system 600,particularly the processor 606, to implement the methods of theinvention according to computer software including instructions.

The computer system 600 described may perform any one of, or anycombination of, the steps of any of the methods according to theinvention. It is also contemplated that the methods according to theinvention may be performed automatically.

The computer system 600 of FIG. 12 is provided only for purposes ofillustration, such that the invention is not limited to this specificembodiment. It is appreciated that a person skilled in the relevant artknows how to program and implement the invention using any computersystem.

The computer system 600 may be a handheld device and include anysmall-sized computer device including, for example, a personal digitalassistant (PDA), smart hand-held computing device, cellular telephone,or a laptop or netbook computer, hand held console or MP3 player,tablet, or similar hand-held computer device, such as an iPad®, iPadTouch® or iPhone®.

FIG. 13 illustrates an exemplary cloud computing system 700 that may bean embodiment of the invention. The cloud computing system 700 includesa plurality of interconnected computing environments. The cloudcomputing system 700 utilizes the resources from various networks as acollective virtual computer, where the services and applications can runindependently from a particular computer or server configuration makinghardware less important.

Specifically, the cloud computing system 700 includes at least oneclient computer 702. The client computer 702 may be any device throughthe use of which a distributed computing environment may be accessed toperform the methods disclosed herein, for example, a traditionalcomputer, portable computer, mobile phone, personal digital assistant,tablet to name a few. The client computer 702 includes memory such asrandom-access memory (RAM), read-only memory (ROM), mass storage device,or any combination thereof. The memory functions as a computer usablestorage medium, otherwise referred to as a computer readable storagemedium, to store and/or access computer software and/or instructions.

The client computer 702 also includes a communications interface, forexample, a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, wired or wireless systems,etc. The communications interface allows communication throughtransferred signals between the client computer 702 and external devicesincluding networks such as the Internet 704 and cloud data center 706.Communication may be implemented using wireless or wired capability suchas cable, fiber optics, a phone line, a cellular phone link, radio wavesor other communication channels.

The client computer 702 establishes communication with the Internet704—specifically to one or more servers—to, in turn, establishcommunication with one or more cloud data centers 706. A cloud datacenter 706 includes one or more networks 710 a, 710 b, 710 c managedthrough a cloud management system 708. Each network 710 a, 710 b, 710 cincludes resource servers 712 a, 712 b, 712 c, respectively. Servers 712a, 712 b, 712 c permit access to a collection of computing resources andcomponents that can be invoked to instantiate a virtual machine,process, or other resource for a limited or defined duration. Forexample, one group of resource servers can host and serve an operatingsystem or components thereof to deliver and instantiate a virtualmachine. Another group of resource servers can accept requests to hostcomputing cycles or processor time, to supply a defined level ofprocessing power for a virtual machine. A further group of resourceservers can host and serve applications to load on an instantiation of avirtual machine, such as an email client, a browser application, amessaging application, or other applications or software.

The cloud management system 708 can comprise a dedicated or centralizedserver and/or other software, hardware, and network tools to communicatewith one or more networks 710 a, 710 b, 710 c, such as the Internet orother public or private network, with all sets of resource servers 712a, 712 b, 712 c. The cloud management system 708 may be configured toquery and identify the computing resources and components managed by theset of resource servers 712 a, 712 b, 712 c needed and available for usein the cloud data center 706. Specifically, the cloud management system708 may be configured to identify the hardware resources and componentssuch as type and amount of processing power, type and amount of memory,type and amount of storage, type, and amount of network bandwidth andthe like, of the set of resource servers 712 a, 712 b, 712 c needed andavailable for use in the cloud data center 706. Likewise, the cloudmanagement system 708 can be configured to identify the softwareresources and components, such as type of Operating System (OS),application programs, and the like, of the set of resource servers 712a, 712 b, 712 c needed and available for use in the cloud data center706.

The invention is also directed to computer products, otherwise referredto as computer program products, to provide software to the cloudcomputing system 700. Computer products store software on any computeruseable medium, known now or in the future. Such software, whenexecuted, may implement the methods according to certain embodiments ofthe invention. Examples of computer useable mediums include, but are notlimited to, primary storage devices (e.g., any type of random accessmemory), secondary storage devices (e.g., hard drives, floppy disks, CDROMS, ZIP disks, tapes, magnetic storage devices, optical storagedevices, Micro-Electro-Mechanical Systems (MEMS), nanotechnologicalstorage device, etc.), and communication mediums (e.g., wired, andwireless communications networks, local area networks, wide areanetworks, intranets, etc.). It is to be appreciated that the embodimentsdescribed herein may be implemented using software, hardware, computercode, or combinations thereof.

The cloud computing system 700 of FIG. 13 is provided only for purposesof illustration and does not limit the invention to this specificembodiment. It is appreciated that a person skilled in the relevant artknows how to program and implement the invention using any computersystem or network architecture.

While the disclosure is susceptible to various modifications andalternative forms, specific exemplary embodiments of the invention havebeen shown by way of example in the drawings and have been described indetail. It should be understood, however, that there is no intent tolimit the disclosure to the particular embodiments disclosed, but on thecontrary, the intention is to cover all modifications, equivalents, andalternatives falling within the scope of the disclosure as defined bythe appended claims.

The invention claimed is:
 1. A computer system method for providing oneor more sensory cue to direct visual attention on a screen display, themethod comprising: (a) defining an area on the screen display whereattention will be measured; (b) programming sensory cues to estimateattention during a programming phase, the programming including:accessing a user interface for defining settings for each sensory cue;entering a minimum time threshold required to accept attention inputdata associated with each sensory cue; selecting parameters for eachsensory cue; choosing sensory cues for participation; and defining aninput device for acquiring attention input data; and (c) executing thesensory cues during an execution phase with respect to a frame orregion, the executing including: (a) initializing each chosen sensorycue for each frame or region to a zero value; (b) recording real-timeattention input data; (c) accepting the real-time attention input dataif the minimum time threshold is met; (d) updating real-time a durationof time between accepted attention input data for each sensory cue ineach frame or region; (e) comparing real-time all durations of time ofall sensory cues within the frames or regions; (f) selecting the frameor region with the longest duration of time thereby demonstrating ameasurement of attention and assigning a neglected status to that frameor region; (g) displaying an activation cue for visual attention in theframe or region on the screen display according to the selectedparameters; and (h) repeating (b)-(g) until the execution phase ends. 2.The computer system method of claim 1 wherein the sensory cue is avisual sensory cue.
 3. The computer system method of claim 2 wherein thevisual sensory cue is a frame component including a window element and aborder element.
 4. The computer system method of claim 2 wherein theselecting includes: selecting a window fill parameter for each visualsensory cue; and selecting a border parameter for each visual sensorycue.
 5. The computer system method of claim 4 wherein the window fillparameter is one or more selected from a group of: a color and anoverlay including opacity, translucency, and transparency.
 6. Thecomputer system method of claim 5, further comprising selecting thecolor using a swatch grid.
 7. The computer system method of claim 5,further comprising selecting the color using one or more models selectedfrom the group comprising another group that includes: hue, saturation,value (HSV) model, hue, saturation, luminance (HSL) model, red, green,blue (RGB) model, and cyan, magenta, yellow, black (CMYK) model.
 8. Thecomputer system method of claim 4 wherein the border parameter is aborder thickness, border style, or combination thereof.
 9. The computersystem method of claim 1 further comprising creating a frame or regionrecord.
 10. The computer system method of claim 9 wherein the creatingincludes: entering an initial location of the frame or region based onx, y coordinates of the screen display; defining an initial size andlocation of each frame or region based on a height dimension and a widthdimension; assigning a color to the frame or region; and adjusting acolor format of the assigned color, wherein the color format includesoverlay, weight value, and weight percentage.
 11. The computer systemmethod of claim 1 wherein the attention input data includes eye gaze ofthe user, eye movement of the user, and device input according tomovement of at least one input device by the user.
 12. The computersystem method of claim 11 wherein the at least one input device includesa computer mouse, touch screen, touch pad, stylus, keyboard, orcombination thereof.